Learning From An Optimization Viewpoint

Optimization has always played a central role in machine learning and advances in the field of optimization and mathematical programming have greatly influenced machine learning models. However the connection between optimization and learning is much deeper : one can phrase statistical and online learning problems directly as corresponding optimization problems. In this dissertation I take this viewpoint and analyze learning problems in both the statistical and online learning frameworks from an optimization perspective. In doing so we develop a deeper understanding of the connections between statistical and online learning and between learning and optimization. The dissertation can roughly be divided into two parts. In the first part we consider the question of learnability and possible learning rates for general statistical and online learning problems without regard to tractability issues. In the second part we restrict ourselves to convex learning problems and address the issue of tractability for both online and statistical learning problems by considering the oracle complexity of these learning problems. I. We first consider the question of learnability and possible learning rates for statistical learning problems under the general learning setting. The notion of learnability was first introduced by Valiant (1984) for the problem of binary classification in the realizable case. Vapnik (1995) introduced the general learning setting as a unifying framework for the general problem of statistical learning from empirical data. In this framework the learner is provided with a sample of instances drawn i.i.d. from some distribution unknown to the learner. The goal of the learner is to pick a hypothesis with low expected loss based on the sample received. The question of learnability is well studied and fully characterized for binary classification using the Vapnik Chervonenkis (VC) theory and for real valued supervised learning problems using the theory of uniform convergence with tools like Rademacher complexity, covering numbers and fat-shattering dimension etc. However we show that for the general learning setting the traditional approach of using uniform convergence theory to characterize learnability fails. Specifically we phrase the learning problem as a stochastic optimization problem and construct an example of a convex problem where Stochastic Approximation (SA) approach provides successful learning guarantee but Empirical Risk Minimization (ERM) (or equivalently Sample Average Approximation (SAA) approach) fails to give any meaningful learning guarantee. This example establishes that for general learning problems the concept of uniform convergence fails to capture learnability and ERM/SAA ap-